Can We Trust Test Outcomes?

Software development is an on-going process. To ensure that the changes made do not break previous functionality, a regression test suite is maintained along with the source code. The correctness and the side effects of code changes are judged based on the output of the tests. However, are these tests’ outcomes always correct? Does a test failure imply a bug and do all tests passing imply a bug free code? The answer is no. Experienced researchers and practitioners know that there is an inherent non-determinism in testing. This paper contributes the first comprehensive study on test outcome non-determinism. We target 153 open-source projects with a novel methodology starting from the commit logs, which enabled us to identify many unreported cases of tests with nondeterministic outcomes (also called flaky tests). We were able to detect 1129 commits from 51 projects that are related to flaky tests, out of which we study 486 commits which are about distinct flaky tests. 61% of these flaky tests do not have associated bug reports. In our study, we focus on 81 commits to analyze in depth. We identify 11 causes of flakiness, provide examples for each, and give recommendations on how to fix them. We also discuss other interesting findings such as the effectiveness of common fixes, the evolution of flaky tests, how hard it is to reproduce a flaky test, and how hard it is to patch it.